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Motivated by the recent discovery of a low temperature spin liquid phase in layered organic 
compound k-(ET)2Cu2(CN)3 which becomes a superconductor under pressure, we examine the 
phase transition of Mott insulating and superconducting (SC) states in a Hubbard-Heisenberg model 
on an anisotropic triangular lattice. We use a renormalized mean field theory to study the Gutzwiller 
projected BCS wavefucntions. The half filled electron system is a Mott insulator at large on-site 
repulsion U , and is a superconductor at a moderate U . The symmetry of the SC state depends on 
the anisotropy, and is gapful with d x 2_ y 2 + id xy symmetry near the isotropic limit and is gapless 
with d x 2_ y 2 symmetry at small anisotropy ratio. 
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Understanding of exotic properties in layered organic 
superconductors k-(ET)2X (X=anion) has attracted 
much interest recent\yl&2££&. In these materials, (ET)2 
dimers are arranged in a quasi-two-dimensional (quasi- 
2D) anisotropic triangular lattice, as schematically shown 
in Fig. 1(a). Since the intradimer hopping integrals are 
much larger than the interdimer ones, and the carrier 
density of the compound is one hole per dimer, the low 
energy electronic structure can be well approximated by 
a 2D Hubbard model at the half filling, where each lat- 
tice site represents a dimer—, as illustrated in Fig. 1(b) 
with nearest neighboring (n.n.) and next n.n. hopping 
integrals t and t' and a strong on-site Coulomb repulsion 
U . Under the pressure, some of the compounds show a 
first-order transition from a Mott insulator to a super- 
conductor. We expect the effect of t' or the geometrical 
frustration to be crucial to the low temperature phases. 
In most of k-(ET) 2 X materials, t'/t ~ 0.4 - 0.8, the ef- 
fect of geometrical frustration is less remarkable so that 
the ground state of the insulating phase is antiferromag- 
netic, similar to the high-T c SC cuprates&2ii2iiiii£. An 
example is the compound with X=Cu[N(CN) 2 ]Br, where 
t'/t ~ 0.6. For systems with a larger t'/t, the frustra- 
tion may destroy magnetic ordering at low temperatures 
and the interplay between the frustration and strong cor- 
relation may lead to the possible emergence of spin liq- 
uid state or unconventional superconductivity. The com- 
pound of X =Cu2(CN) 3 (t'/t ~ 1.06) appears to be an 
excellent candidate for spin liquid state without manifes- 
tation of any magnetic ordering as indicated in the NMR 
at low temperatures down to 32 mKi£, reminiscent of the 
resonating valence bond state proposed by Andersoni^i^ 
in a triangular lattice system. The compound becomes 
SC under moderate hydrostatic pressure with the maxi- 
mum T c of 3.9 K. 

The SC pairing symmetry in the layered organic 
conductors remains controversial. Experimentally, the 
NMR measurements2ii2ii&, the angular dependent scan- 
ning tunneling microscopyii and the thermal conductiv- 
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FIG. 1: (a) k — type ET molecular arrangement within the 
conducting plane, (b) Anisotropic triangular lattice struc- 
ture of the 2D Hubbard-Heisenberg (t-t'-J-J'-U) model. Also 
shown are three unit vectors on the lattice. 



ity measurements in the vortex stated all suggest the 
existence of nodes in the gap and indicate a spin sin- 
glet pairing d x 2_ y 2 symmetry, similar to that of high-T c 
cuprates. However, the temperature dependence of the 
penetration depthi2iS£ and of the specific healiSi seem to 
support a full gapped SC state. Note that there are also 
results consistent with a gap with node a 22 ' 2 ^ . On the 
theoretical side, fluctuation exchange approximation for 
the Hubbard model shows a pure d x 2_ y 2-w&ve symme- 
trjiSi while the variational Monte Carlo and mean field 
study for a generalized t-J model (t-t'-J-J' model) sug- 
gest that d x 2_ y 2 + id xy - (d + id-) wave state is more sta- 
bl o 2 ^i 2 ^i 2 7 . Concerning the order of SC to Mott insula- 
tor transition, there are two recent studies^SS based on 
variational Monte Carlo calculation by analyzing a 2D 
Hubbard model. They find a first-order SC to Mott in- 
sulator transition for intermediate U and the SC state is 
restricted to the dj;2_ y 2-wave symmetry. 

In a previous paper, we considered the competition 
between d a ,2_ y 2-wave SC and antiferromagnetic states 
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in k-(ET)2 salts in the parameter region t'/t < 0.8 30 . 
The Gutzwiller renormalized mean-field theory 31 was 
employed to study an anisotropic Hubbard-Heisenberg 
model in a 2D lattice. We found a first-order transi- 
tion between a Gossamer superconductor at a smaller U 
and an antiferromagnetic insulator at a larger U. Our 
result was qualitatively consistent with major experi- 
ments. In this paper, we are primarily interested in 
layered organic conductor with t'/t ~ 1, with the com- 
pound X =Cu 2 (CN) 3 in mind. Our focus will be on the 
Mott insulator to SC transition and the pairing symme- 
try of the SC state. We expect the anisotropic ratio t'/t 
will affect the pairing symmetry. Since the compound 
of X =Cu2(CN)3 is not magnetically long range ordered, 
we shall neglect the antiferromagnetism in our study. We 
find that at half filling, the ground state is a metal for 
small U, a Mott insulator for large U, and a Gossamer 
superconductor at intermediate U, whose pairing is d+id 
near the isotropic point. A transition between different 
pairing symmetries may show up by tuning the model 
parameters. 

We start with a Hubbard-Heisenberg model on an 
anisotropic triangular lattice, 

H = U ^n iT n a - M C L C ^ + h - c -) 

i (y)<7 
+ ^JijSi ■ Sj - fl^Uj, (1) 



where c ia is to create a hole with spin a at site i, Si is 
a spin operator, fi is the chemical potential, and (ij) de- 



notes a neighboring pair on the lattice. l 13 



tii t, Ji' 



J 



for the neighboring pairs along x and y directions, and 
tij = t', Jij = J' for the neighboring pairs along the a- 
direction as shown in Fig. 1(b). Hereafter we set t = 1 
and use t as the energy unit, and choose J/t = 1/3 
and consider mainly the system in the parameter range 
t'/t = 0.8 - 1.2. We choose J' /J = (t'/t) 2 , consistent 
with the relation of J = 4t 2 /U in the large U limit. 
We will exclusively consider the half filled case. When 
t'/t = J' /J = 1, the model becomes isotropic. At 
U — ► oo, the model is reduced to the Heisenberg model. 
We remark that the Heisenberg model on an isotropic tri- 
angular lattice has a long range antiferromagnetic order 
with an angle of 2-7r/3 between neighboring spins. How- 
ever, there is the possibility to have a spin liquid state in 
the anisotropic Heisenberg model 3 ^ and in the anisotropic 
Hubbard model 3 ^. In what follows below, we will devote 
to the non-magnetic states in our variational approach. 

We consider a partially Gutzwiller projected BCS trial 
wavefunction 34 i 35 i 36 , 

l*GS> = IIP^BCS) 

i 

I^bcs) = H(u k +v k c{ lC l kl )\0) (2) 

k 

where Pi = z ni (l — an%^7iii) is a projection operator to 
partially project out the doubly occupied electron states 



on the site i and < a < 1 measures the strength of the 
projection, z is the fugacity to ensure the charge density 
is unchanged by projection 3 ^ 3 ^ 3 ^ 3 ^. Note that introduc- 
tion of the fugacity does not change the previous results 
on the renormalized mean field theory for the Gutzwiller 
state for that the number of charge carriers were assumed 
to be the same^i. 

We then apply the Gutzwiller approximation to esti- 
mate the variational energy. In this scheme, the effect of 
the projection operator is taken into account by a set of 
renormalized factors, which are determined by statistical 
countings^S^i. Let (Q) be the expectation value of Q in 
the state \^gs) and (Q)o be that in the state \^bcs)- 
The Gutzwiller approximation gives, 

( c L c j<?) ~ 9t{c\ a c ja ) , (Si ■ Sj) w g s (S t ■ Sj) (3) 

where gt and g s are the Gutzwiller renormalized coeffi- 
cients and can be derived as follows 3 ^, 



9t 



(n-2d)(y/d+y/l-n + df 

(1 - n/2)n 
r (n-2d) l2 



L (l - n/2)n' 



(4) 



where n is the average electron number and d = (n^riii) 
is the average electron double occupation number. There 
is one-to-one correspondence between a and d. Within 
the Gutzwiller approximation, we have, 



(l-af 



d(l-n + d) 
(n/2-d) 2 



(5) 



In terms of the renormalized coefficients, we can get the 
effective Hamiltonian, 



H, 



eff 



Ud- g t > UicLc 



h.c. 



g s ^ Jij Si ■ Sj 

{ij) 



fi 2_^ni 



(6) 



The variation of the projected state |^gs) f° r H in (1) 
is thus reduced to the variation of the unprojected state 
\^bcs) for H e ff. Note that the Gutzwiller projection on 
a BCS state may change the average number of electrons 
of the state. 

To proceed further, we introduce two types of mean 
fields, (t = ±x,±y,±a) 



Ai,,+ r = (cj4.Cj+ T f - Ci]Ci +T i)a 

Xi,i+T = / / \ 



(7) 



We consider the translational invariant state with even 



parity, A* 



A r , andxM+r = Xr- Applying 



the mean field theory to H e f f , we obtain, 

kl 2 = i E k-H)/ 2E » 



E % = 



4 



(8) 
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where 
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2t T cos k T — jU ^ J T Xr cos fc 



F k = 4-9« X! Ar cos kr 



(9) 



where k T — k ■ f, and the sum in r runs over the three 
unit vectors. In particular, we assume A a to be real, and 
A x and A y to have a phase 8 and —8 relative to A a , 
respectively 



A a = |A a |, A x = \A 3 



-iO 



Ay=\Ay\ 



(10) 



The case of A a = and 9 — tt/2 corresponds to the pure 
d x 2_ y 2 state. In the case 9 = 0, we have an extended s 
wave state. In general, A a ^ and 9 ^ tt/2, we have 
d + id symmetry^. 

The self-consistent equations are given as 



k 

-^J2 cosk - £ k/ E k 



Xr = 



6 = jfEei/Bi 



(11) 



where the last equation is on the number of holes which 
is zero at the half filled of our interest here, and N is the 
number of total lattice sites and the sum in A; runs over 
the first Brillouin zone. The ground state energy is 



E =Ud— 2g t £ UXr -%£ J ^ A - 



\Xr 



•(12) 



All the mean fields are determined self-consistently for 
each set of model parameters U , t'/t and a given vari- 
ational parameter d, from which we obtain the lowest 
energy state with respect to the variation of d. 

We first examine the evolution of the ground state by 
varying the anisotropy ratio t' jt. The anisotropy ratio 
is a measure of the geometrical frustration, which mod- 
ifies the shape of Fermi surface and therefore plays an 
important role in determining the competition between 
Mott insulator and various SC states. As shown in our 
previous study^S, for small anisotropy t'/t, the ground 
state is an antiferromagnetic insulator at large U and a 
Gossamer superconductor at medium U, followed by a 
normal metallic state at small U, and the transition be- 
tween Mott insulator and SC state belongs to the first 
order. We expect the physics of the model in a trian- 
gular lattice with large anisotropy may be qualitatively 
different from that of a less anisotropic lattice. For large 
t'/t, the antiferromagnetic state may be degraded into 
spin liquid state and time-reversal symmetry broken SC 
state may show up as well. We note that the transition 
from the Mott insulator to SC state remains first order. 
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FIG. 2: The relative phase of pairing parameter as a function 
of t'/t. 



In our calculation, the relative phase 9 of pairing pa- 
rameter is obtained by the minimization of the total en- 
ergy. Due to the numerical difficulty to determine the 
exact value of 9 for small but finite d, we present the 
results in Fig. 2 for the case d = 0. The approximate 
critical value of (t'/t) c ~ 0.81 distinguishes the different 
pairing symmetry regionlS, the SC order is always with 
d x 2_ y 2 symmetry (9— tt/2) for t'/t < (t'/t) c while imagi- 
nary part of pairing parameter emerges and the relative 
phase 8 exceeds tt/2 for t'/t > (t' /t) c . The SC order 
parameter with nonzero imaginary part corresponds to 
d + id-wave state which breaks time-reversal symmetry. 
The presence of such state may result in the appearance 
of excitation gaps all over the Fermi surface. This time- 
reversal symmetry broken state might be detectable by 
using muon-spin rotation measurements^. It is interest- 
ing to note that when t' = t and J' — J, the relative 
phase 9 is equal to 27r/3 where all the three amplitudes 
become symmetrical and we have A=A a; =A, / =A a and 
X —Xx—Xy—Xa- This special pairing symmetry is due 
to the geometrical symmetry of the isotropic triangular 
lattice. This result is fully consistent with other works 
for a, t-J model on a triangular lattic e - 26 ! 27 . The contin- 
uous change of the relative phase 8 is found by further 
increasing the anisotropic parameter t'/t. For t'/t > 1.3, 
we find that the value of 9 saturates to 37r/4. 

Our detailed results for t'/t = 1 are presented in 
Fig. 3. Here we denote the SC order parameter as 
Asc = ( c ii c iT — c ij c jl)- Within the Gutzwiller approxi- 
mation, Asc can be expressed as A$c = 9t A. The effect 
of large U may suppress the double occupancy d and sta- 
bilize the Mott insulating state. As shown in Fig. 3(a), 
it is obvious that there exists a critical J7 C (~ \2.2t) to 
separate a SC state at a small U from a Mott insulator 
at a large U, and the transition is first order with no co- 
existence of the two phases. At U < U c , both A and Asc 
increase monotonically as U increases. Asc is smaller 
than A because of the prefactor g t < 1. At the critical 
point, A increases to its maximum while Asc jumps to 
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FIG. 3: Pairing amplitude A x and its SC order parameter 
Af c (a), and electron double occupancy number d and the 
projection parameter a (b), as functions of U/t for t'/t = 1. 
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FIG. 5: Phase diagram in parameter space t'/t and U/t. 



zero suddenly. At U > U c , the system becomes an insu- 
lator with d = 0, or indicates the full projection in the 
insulating phase . The projection parameter a reaches 
its maximum in the SC phase followed by a discontinu- 
ous jump to 1 at U = U c . All these features suggest that 
the first order transition is robust. 

Another interesting feature reveals that in the SC 
phase just below U c , the double occupation number d 
is quite small (~ 0.04), while a is large (~ 0.91). In 
accordance with Laughlin's proposal of Gossamer super- 
conductivitj^, very tiny superfluidity density appears 
in the SC state. It is obvious that the SC state in this 
system can be well characterized by the Gossamer su- 
perconductor. All the previous studies for less frustrated 
system show the presence of appreciable (not so tiny) su- 
pcrfluid density. In our present study, it is clear that the 
effect of strong geometrically frustration may remarkably 
suppress the superfluid density of the SC stated. 




FIG. 4: Pairing amplitudes and its SC order parameter as 
functions of U/t for t'/t = 1.06. (a) and Af c . (b) A a 
and Af c . 



Next we study the particular case for the compound k- 
(ET)2Cu2(CN)3 which has a large degree of geometrical 
frustration t'/t ~ 1.06. One expects the general features 
are similar to the t'/t = 1 case. As depicted in Fig. 4, we 
notice two features which are in contrast to the t'/t = 1 
case. For the anisotropic parameter t'/t > 1, the ampli- 
tude of A becomes larger than that of A^ while the SC 
order parameter of Af c is more pronounced than that 
of Af c . In other words, the relative phase of SC or- 
der parameter 9 may deviate from the symmetrical value 
2tt/3 and will be around 0.707T. Another distinctive fea- 
ture shows that the Mott transition point U c strongly 
depends on lattice structure anisotropy t'/t. Due to the 
competing nature between Mott insulating phase and 
the SC phase, the increasing of geometrical frustration 
t'/t may eventually destabilize the Mott insulating phase 
which may lead to the Mott transition and result in a SC 
phase. We find that the critical value increases slightly 
from U c = 12.2 for t'/t = 1 to U c = 12.5 for t'/t = 1.06. 
This result suggests that the increasing oit'/t may effec- 
tively enhance the critical value U c . 

Fig. 5 displays the phase diagram in the parameter 
space of t'/t and U/t. It is technically difficult to de- 
termine precisely the critical boundaries between various 
phases due to the presence of degenerate states at half 
filling for highly frustrated system. In practice, we define 
a paramagnetic metallic phase if both A^ and A a are less 
than 0.01 and a d + id state if A Q > 0.01. At very small 
A, the energy discrepancy between these phases becomes 
indistinguishable in our calculation. We use dashed lines 
to represent the phase boundaries with very small en- 
ergy differences. The solid line denotes the first-order 
phase boundary between SC phase and Mott insulating 
state. According to our phase diagram, there exists four 
distinct phases. The system is in the Mott insulating 
phase at large U and small t' , the paramagnetic metallic 
phase at small U and large t' , and the two possible SC 
phases at the intermediate parameter region. For small 
t'/t, the phase diagram is consistent with our previous 
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study^S. Note here that the time-reversal symmetry bro- 
ken d + id state is preferable for large t'/t while the pure 
d a ,2_ !/ 2-wave state is more stable for small t'/t. Since the 
effect of pressure may increase U/t and/or to increase 
t'/t, it is quite plausible that a flow of parameters U/t 
and t'/t may lead to a pairing symmetry transition from 
d x 2 +y 2 to d + id state. In a very recent papei4^, the 
transition from d + id to d-wave is also indicated. Two 
recent investigationsS&SSi addressed the first-order Mott 
transition by applying variational Monte Carlo method 
for the layered organic superconductors by analyzing a 
two-dimensional Hubbard model. By using a modified 
variational wavefunction with doublon-holon binding fac- 
tors, they found an unconventional SC ground state for 
medium U, sandwiched between a normal metal at weak 
coupling and a spin liquid at larger coupling. A first- 
order Mott transition takes place at certain value of U. 
Our renormalized mean field theory can take into account 
approximately the effect of Gutzwiller projection and the 
results agree qualitatively with theirs. It is well known 
that the Brinkman-Rice 46 * type metal- insulator transition 
for Hubbard model belongs to the second-order. It seems 
to us that the extra spin exchange term in the Hubbard- 
Heisenberg model makes much difference and changes the 



Gossamer SC to Mott insulator transition to be first- 
order. 

In summary, we have utilized the Gutzwiller mean-field 
method to study the t-t'-J-J'-U model on a highly frus- 
trated anisotropic triangular lattice. We find that the 
pure d x 2_ y 2 state with nodal structure is stable if the ge- 
ometrical frustration parameter t'/t < (f /t) c while the 
gapped d + id state is more preferable for t'/t > (t'/t) c in 
the SC phase. In the case oft'/t = 1, the relative phase 
of the SC order parameter is equal to 27r/3 which reflects 
the geometrical symmetry of isotropic triangular lattice. 
For a fixed t'/t, the ground state is found to be a para- 
magnetic metal at small on-site Coulomb repulsion U; a 
Mott insulator at large U; and a superconductor at inter- 
mediate U . This SC state can be well described by the 
Gossamer superconductivity with tiny superfluid density. 
The transition between SC and insulating phases belongs 
to first order. Our results may shed light on the un- 
derstanding of the exotic features of k-(ET) 2 Cu2(CN) 3 , 
which undergoes a possible spin liquid to superconductor 
transition at half filling under pressure. 

This work is supported by the RGC grants of Hong 
Kong SAR government, and seed funding grant from the 
University of Hong Kong. 
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